Lecture notes for “Analysis of Algorithms”: Markov decision processes

We give an introduction to infinite-horizon Markov decision processes (MDPs) with finite sets of states and actions. We focus primarily on discounted MDPs for which we present Shapley’s (1953) value iteration algorithm and Howard’s (1960) policy iteration algorithm. We also give a short introduction to discounted turn-based stochastic games, a 2-player generalization of MDPs. Finally, we give a short introduction to two alternative criteria for optimality: average cost and total cost. The presentation given in these lecture notes is based on [6, 9, 5]. 1 Markov decision processes A Markov decision process (MDP) is composed of a finite set of states, and for each state a finite, non-empty set of actions. In each time unit, the MDP is in exactly one of the states. A controller must choose one of the actions associated with the current state. Using an action a ∈ A incurs an immediate cost, and results in a probabilistic transition to a new state according to a probability distribution that depends on the action. The process goes on indefinitely. The goal of the controller is to minimize the incurred costs according to some criterion. We will later define three optimality criteria: discounted cost, average cost, and total cost. Formally, a Markov decision process is defined as follows. We use ∆(S) to denote the set of probability distributions over elements of a set S. Definition 1.1 (Markov decision process) A Markov decision process (MDP) is a tuple M = (S,A, s, c, p), where ∗School of Computer Science, Tel Aviv University, Tel Aviv 69978, Israel. E-mail: [email protected].